PyTorch tutorial: LSTM

LSTMはRNNの拡張という理解で読み始めた

『ゼロから作るDeep Learning②』を読んだ記憶

In the case of an LSTM, for each element in the sequence, there is a corresponding hidden state h_t, which in principle can contain information from arbitrary points earlier in the sequence.

系列の各要素に対応する隠れ層

隠れ層は原則として、系列における任意の先行する点の情報を含む

（系列の今の要素よりも前の情報が使える）

Pytorch’s LSTM expects all of its inputs to be 3D tensors.

The first axis is the sequence itself, the second indexes instances in the mini-batch, and the third indexes elements of the input.

1st, 2nd axis (dimension) はサイズ1とする

2nd: ミニバッチは考えない（？）

1st: 一度に1系列進める

code:python

>> import torch

>> import torch.nn as nn

>> torch.manual_seed(1)

>> lstm = nn.LSTM(3, 3)

>> hidden = (torch.randn(1, 1, 3), torch.randn(1, 1, 3))

>> for i in inputs:

... print(i)

... out, hidden = lstm(i.view(1, 1, -1), hidden)

... print(out)

... print(hidden)

...

code:python

>> torch.cat(inputs).size()

>> inputs = torch.cat(inputs).view(len(inputs), 1, -1) # Add the extra 2nd dimension

>> inputs.size()

>> hidden = (torch.randn(1, 1, 3), torch.randn(1, 1, 3))

>> out, hidden = lstm(inputs, hidden)

>> print(out)

>> print(hidden)

the first value returned by LSTM is all of the hidden states throughout the sequence.

the second is just the most recent hidden state (compare the last slice of "out" with "hidden" below, they are the same)

上の例でhiddenのインデックス0はoutと一致

"out" will give you access to all hidden states in the sequence

"hidden" will allow you to continue the sequence and backpropagate, by passing it as an argument to the lstm at a later time